Wip/modeling commons/patch in place - #61
Open
omargfh wants to merge 17 commits into
Open
Conversation
- Move file-key building, .nlogo parsing, legacy row access, dedupe and the per-node migration routine out of archive.ts so patch.ts can reuse them. - Cover the extracted code with unit tests, including the exact write sequence createModelFromNode emits against a recording transaction. - Fix parseNlogox: it called DOMParser, which Node does not have, so every .nlogox model silently stored a null netlogoVersion and infoTab. Replaced with a dependency-free parser. Already-migrated rows still hold nulls. - Pin dob to UTC midnight. pg reads a `date` as local midnight while Prisma stores @db.Date from the UTC day, so a birthdate never compared equal to itself and shifted by a day in positive UTC offsets. - Replace the shell-based avatar copy with fs.copyFile; it interpolated a legacy filename into a command string. - Type-check prisma/lib and the migration scripts.
archive.ts only skips rows that already carry a legacyId, so it cannot see edits, deletions or new children of an already-migrated node. patch.ts applies those from a diffdb.sh diff. - Materialises the whole change set as a plan before writing, so the dry run and the real run compute the same thing. Applying replays it in one transaction and writes an expectations manifest for --verify-only. - Trusts only `side` and `id` from the diff and re-reads rows from the legacy snapshot, since diffdb.sh md5s the `contents` columns. Deleted rows are the exception; their row_to_json is the only surviving copy. - Addresses everything by legacyId. No existing row's id column is touched. - Recomputes previews from the whole attachment set rather than patching them: a preview is one column on the latest version and the highest legacy attachment id wins it. - Moves tags and the preview onto an appended version, matching archive.ts, which keeps them on the latest version only. - Leaves updatedAt alone; Prisma's @updatedat overrode it on create, so no migrated row ever held the legacy value. - Refuses to guess on mid-sequence version deletes, out-of-order appends and in-place attachment edits, none of which can be mapped back safely.
Builds a legacy database with a migrated baseline and an incoming schema carrying sixteen change scenarios, archives the baseline, diffs, patches, then archives the incoming snapshot into a second database and compares canonical uuid-free dumps of both. Asserts that archive(baseline) + patch(diff) equals archive(new snapshot), with the soft-deleted model checked separately since a fresh archive never sees it.
A node that gained a version and lost its last preview attachment in the same diff kept the old preview: the resync compared against the database row for the appended version, which does not exist yet, read null, and concluded a cleared preview was already applied. The append then carried the dead key forward. Compare against what the append will carry instead, and clear that carry when a resync supersedes it. Covered by a rehearsal scenario; the script's own verify passed this because the manifest was built from the same wrong plan.
From a review pass over the patch script. - Derive a version's object key from its legacy id instead of a fresh uuid, and diff the row before writing. A modified version previously uploaded a duplicate object and repointed netlogoFileKey on every run, so re-applying the same diff was not the no-op it was documented to be. - Refuse an ambiguous ModelAdditionalFile match rather than deleting whichever row came back first. Two attachments on one model sharing a filename, or a null created_at, could otherwise delete the wrong file irrecoverably. - Drop a tag from the resolution map once it is scheduled for deletion, so a tagging in the same diff is not created and then cascade-deleted. - Assert the appended version's preview in the manifest; only carried tags were covered, so a wrong carried preview could pass verification. - Strip every uuid segment when comparing preview keys, not just the first, which was the model uuid and identical on both sides. Rehearsal now covers an in-place version edit.
…gacy-migration - rename archive.ts, patch.ts and archive-upload.ts to initial-import.ts, apply-diff.ts and upload-files.ts, naming the job rather than the source - move the staged output and avatar snapshot under the same directory - run db:legacy:patch through tsx directly; wrapping it in `sh -c` swallowed the --apply / --verify-only / --skip-upload flags it reads from argv
- ModelAuthor.collaboratorType carries legacy collaborator_types.name - NonMemberContributor archives credited people who never held an account Both are provenance only; no business logic reads either.
initial-import.ts derived ModelAuthor from version history alone, so a collaborator who never uploaded a version had no row at all. - members fold into ModelAuthor as role=contributor, which grants write access since canWrite treats any contributor as a writer - non-members are archived in NonMemberContributor, unread by the app - idempotent: keyed on the existing (modelId, userId) rows and on NonMemberContributor.legacyId, so a second run reports nothing to do
migrateInteractions tracked its per-table count locally and logged it, but never wrote it back to the report, so report.json always claimed zero. The docs took that at face value: the phase writes ~6.2M rows on the production snapshot. Also records the full-scale rehearsal and the recompute step, which nothing referenced even though the import leaves the denormalized Model counters at 0.
sessions (1.3M rows) and ip_locations (120k) map to nothing in the new schema, so apply-diff.ts always discarded their diffs. Scanning them was most of the script's runtime. Output is byte-identical without them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.